Virtual Tree: A Robust Overlay Network for Ensuring Interval Valid Queries in Dynamic Distributed Systems

Today’s large scale distributed systems are characterized by strong dynamics caused by the inherent unreliability of their constituting elements (e.g. process and link failures, processes joining or leaving the system). This continuous dynamism has a strong negative impact on distributed algorithms designed to work on such systems. Regular registers [1], Replication [2], in-network aggregation[3], are all examples of such problem. Considering the in-network aggregate query answering problem in large scale dynamic systems, a precise semantics was introduced by Bawa et al. in their seminal work [4]. One of the semantics introduced in that work, namely Interval Validity (IV), requires the answer to contain at least contributions from all the processes that remain in the system from the moment the query is issued, until the last answer is collected. This kind of semantics plays a fundamental role in many applications as it prevents contributions coming from correct processes, remaining inside the system, to be eclipsed by transient errors and failures. The same work also proved the impossibility of enforcing interval validity as long as churn (i.e. the rate at which processes can crash/leave or join the system) is unbounded. However, practical experience shows that many systems undergo a continuous (in time) but limited (in its strength) level of churn, and that this level can be reasonably predicted by analytical assessment or by direct measurements [5]. By exploiting this aspect, it would be thus possible to circumvent the previously cited impossibility result and provide interval valid answers to in-network aggregate queries. On the basis of these motivations, this paper presents the first solution for in-network aggregate query processing that is able to provide query answers complying with the interval validity semantics in large-scale dynamic systems with bounded churn. The correctness of our approach is supported by both formal proofs and an extensive set of simulation-based experiments that, due to space constraints, are reported in a separate technical report [6].